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A drowsiness and fatigue problems among the drivers are the main factor 
that contributes to road accidents. These problems are vital to be resolved as 
they could contribute to damage of road facilities, vehicles and most 
importantly the loss of lives. In avoiding these matters, a proper mechanism 
is needed to alert the driver to stay awake throughout the driving journey. 
Thus, this study proposed a real-time prototype for recognizing the 
drowsiness and fatigue face expression of the driver. The methodology of 
this study involves facial features detection using Viola-Jones algorithm to 
detect the exact position of both left and right eyes and mouth. Next, based 
on the detected eyes and mouth beforehand, the segmentation processes 
performed on both eyes and mouth using Sobel edge detection to obtain 
facial regions. The feature extraction phase is conducted using shape-based 
feature to obtain the extraction values. Support vector machine (SVM) 
classifier is deployed for the recognition task. A total of 100 images are used 
during the testing stages. This study achieved a competetive result of 90.00% 
of accuracy. Yet, hybridization or integration of more image processing 
techniques will be performed in the future to improve the current accuracy 
obtained. 
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1. INTRODUCTION 


Thousands of Malaysians lose their lives continuously due to the road accidents [1]. Road accidents 
are a situation where it involves a collision between the vehicles in an undesirable or unexpected event 
without an intentional cause and plan [2]. Malaysians are not aware that every day they got a lot of 
tendencies of an accident whether they are the driver or passenger. The driver is the main actor in this 
situation in keeping the passenger safe. 

There are several causes that lead to accidents due to negligent driver such as drowsiness, 
unconscious driver, exhaustness, lack of sleep, or involve in a long drive without short rest [3-4]. Besides, 
having enough sleep is important too. The most common average adult clock is seven to eight hours per 
night. Recent studies outlined by [5] suggested that staying up late at night, consuming excessive caffeine 
and insomnia may contribute to drowsiness. The term “drowsy” is synonymous with sleepy, which simply 
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means an inclination to fall asleep. The brain may start to give instruction to hibernate in getting enough 
sleep at any time [6]. 

In another note, numerous methods were implemented to measure the driver’s symptoms when 
experiencing dowsiness and fatigue while driving [7-10]. Many researcher all over the world has also agreed 
that a common characteristics shared by a drowsy and fatigue driver is based on their body attitude and the 
facial expression [11]. Kitajima’s facial expression estimation method is one of the established method in 
measuring the drowsy and fatigue driver facial expression [12]. The method has outlined the characteristics 
in recognizing a drowsiness level up to five level. 

Accordingly, the use automation system with less or no human interference is beneficial for human 
specifically in detecting and recognizing the driver’s drowsy and fatigue face expression [13]. The 
implementation of image processing in describing the drowsy and fatigue facial expression can lead to the 
detection and recognition of the driver’s drowsy and fatigue expression automatically and effectively [14-17]. 

Hence, this study proposed a real-time drowsiness and fatigue facial expression recognition using 
image processing technique. The detection of facial features is done using Viola-Jones algorithm in detecting 
the exact position of both left and right eyes, and mouth. Next, a segmentation process is performed to both 
eyes and mouth using Sobel edge detection. The shape-based feature extraction is then conducted to analyse 
the characteristics of the segmented eyes and mouth regions. Finally, a technique of support vector machine 
(SVM) is deployed for the drowsy and fatigue recognition task. 

This paper is divided into five sections. The first section consists of the introduction and research 
motivation. Second section comprises of the research methods adapted in this research. Furthermore, section 
three entails the analysis and findings of this research. Eventually, the last section summarizes the research 
findings respected to research objection, as well as recommendations for future research. 


2. RESEARCH METHOD 

The aim of thos study is to recognize drowsy and fatigue face expression of the driver using SVM 
technique and to evaluate the performance of the recognition using confusion matrix. Figure | depicts the 
proposed flowchart of this study. 


Data collection 


Processing 


Face detection using Viola Jones algorithm 
Eyes and mouth segmentation using Sobel 


Features extraction using shape-based features 


Classification using support vector machine 


Figure 1. Proposed flowchart for this study 


The proposed flowchart of this study begins with data collection. Next, the processing phase 
consists of four sub-processes which are face detection, eyes and mouth segmentation, feature extraction, and 
classification. The face detection is used to recognize the exact position of both left and right eyes, and 
mouth. In narrowing the processing area, the detected region of left and right eyes, and also mouth are then 
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segmented. A shape-based feature extraction is used to study the characteristics of each segmented region. 
Next, the drowsy and fatigue recognition is performed using the SVM technique which produces the final 
outcome of the classification subsequently. The detail explanation of each process involved is elaborated 
further in the next subsection. 


2.1. Data collection 

The data collection was conducted in Universiti Technology Mara (UiTM) Cawangan Melaka 
(Kampus Jasin). Two female students imitated various facial expression includes drowsy and fatigue facial 
expression. The length of the captured videos is in the range of 30 to 60 seconds. Total of 400 images of left 
eyes, 400 images of right eyes and 400 images of mouth are generated from these videos. 

Few numbers of videos composed of person imitates the drowsy and fatigue expressions are 
recorded. The captured videos are separated into collection of still images. The images are then used in the 
processing stage. In general, a drowsy and fatigue driver will blink eyes slowy or rather closed the eyes as 
well the mouth will yawn. Figure 2 shows example of facial expression showing drowsy and fatigue state. 

The classification of facial expression is adapted from the Kitajama’s facial expression method [12]. 
It classifies the driver’s facial expression based on certain behaviors. Table 1 tabulates the description of the 
Kitajama’s facial expression method. 


Table 1. Description of Kitajima’s facial expression method [12] 


Drowsiness Description Behavior 
Level 
1 Not Sleepy Eyes move quickly and motion is active. 
2 Slightly Sleepy | Eyes move slow slightly and lip opens a little. 
3 Sleepy Mouth moves, touches the face and reseating 
4 Rather Sleepy —_ Head is shaking, frequent yawning and blinks 
are slow. 
5 Very Sleepy Eyes are closed and head falls backward. 


Figure 2. A person showed drowsy 
and fatigue facial expression with 
closed eyes and opened mouth 


2.2. Face detection using Viola-Jones 

From the image generated, face detection is then performed. It is one of the important steps for 
further detecting drowsy and fatigue face expression. This step is crucial as it needs to locate the exact 
position of the face. As subsequently, the location of both right and left eyes and mouth will be determined. 
Viola-Jones algorithm is a local feature technique which categorized as a feature-based technique [18]. 
Therefore, this study deployed Viola Jones algorithm in detectiong the face as well the location of both right 
and left eyes and the mouth of the driver. 

The Viola-Jones algorithm uses Haar-like features [19] and the first step in this algorithm is to 
convert the input image into an integral image. Integral image is a summed area table for the purpose to 
speed up the computation of the sum values in a rectangle subset of the pixel grid. Equation (1) denotes the 
construction of the integral image, where the integral image at location x, y contains the sum of the pixels 
above and to the left of x, y position [20]. 


ii(x,y) = Dictenaley UX oD (1) 


Next, after the sum of the rectangular area is computed, the feature associated with pattern P of 
image J is defined by (2) as follows. This will allow the comparison between the pattern and image. 
Equation (2) shows the implementation. 


Yasisw Lasjen (L) 1g, pis write = Lisisn Lasiew! GS) 1eq, pris ptack (2) 


In (2), J and P denote an image and a pattern respectively. Both of image, J and pattern, 
P in the same size of N x N. The integral images obtained from (1) will allow integrals for the Haar-like 
features to be calculated. To overcome the issues in different lightening condition, all images will be 
normalized using mean and variance [19]. Figure 3 shows the five Haar-like patterns used to describe the 
position and pattern of the facial features. 
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On the other hand, Figure 4 illustrates the sample detection of face, eyes and mouth using Viola- 
Jones algorithm. The five derived patterns are considered the facial features on face [21]. In the 
implementation, the extracted features to represent the facial features are the horizontal, vertical and 
horizontal with spaces and checkered [20]. Hence the use of Haar-like features patterns is justified. This 
Viola-Jones algorithm will effectively detect the face first and detect the position of the eyes and mouth 
subsequently. Next, the facial features detected need to be segmented. 


to = 


Figure 3. Five Haar-like features patterns to describe 
the facial features [21] 


Figure 4. Graphical representation of face 
detection using Viola Jones algorithm 


2.3. Eyes and mouth segementation using sobel 

In segmentation phase, Sobel edge detection is used for segmenting eyes and mouth on the driver’s 
face detected using Viola-Jone algorithm earlier. Sobel edge detection is suit for its high frequency variation 
as eye blinking and mouth yawning. It works by computing the gradient of image’s intensity at each pixel in 
the image using two different 3 x 3 matrix kernels. Each kernel constitutes of x-direction kernel, Gx and the 
y-direction kernel, Gy. Figure 5 shows the kernel used in the Sobel edge detection while Figure 6 depicts the 
segmentation of eyes and mouth using Sobel edge detection. 


Sobel edge applied. Sobel edge applied. 


Horizontal Vertical 


Figure 5. Kernel used in the 
Sobel edge detection 


Figure 6. Segementation of eyes and mouth using Sobel edge detection 


2.4. Features extraction using shape-based features 

Features extraction is conducted next to extract the useful information from the images for the 
classification purpose. This study uses eyes and mouth features that have been segmented using Sobel edge 
detection. Eyes and mouth features extraction is performed using shape features to obtain the feature vectors 
that will be used in the classification later. The list of feature vectors calculated are area, perimeter, 
equivalance diameter, major axis and minor axis. These features are selected due to is suitability in 
representing the state of eyes and mouth. Regionprops() in Matlab is used for extracting these features. 
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2.5. Classification using support vector machine 

This section discusses on the classification using the state-of-the-art support vector machine (SVM) 
classifier. SVM classifier is a very useful machine learning tool [22-23]. In this stage classification using 
SVM is a process to determine whether the driver is in drowsy or fatigue condition based on the set of images 
obtained from the video. The implementation of SVM classifier involes the use of trainImageClassifier() 
function [24] where two different category of classes are build. The classes are either the driver is drowsy 
and fatigue, or the driver is awake. The model is trained and the support vectors (SVs) for these classes are 
generated. Hence, the testing can be conducted based on the trained model built. Figure 7 illustrates example 
of kernel used in SVM. 


(‘Decision Boundary-’, ‘linear’] (‘Decision Boundary:", ‘poly'] [‘Decision Boundary:’, 'rbf'] 
~ * ” F Z  — a = 


Figure 7. Example of kernel SVM 


The prediction of drowsy and fatigue face expression on the driver’s face are based on the both 
close left and right eye and open/closed mouth. These symphtoms indicates the driver is yawning and the 
eyes are shut. Subsequently, if the frames from the vidoe show the true predicted signs of drowsy and fatigue 
state continuously in 3 seconds and above, the alarm will be turned on [3]. This alarm is aim to wake the 
driver up from feeling sleepy, to be more cautious and keep focus while driving. 


2.6. Evaluation 

The proposed study is evaluated using confusion matrix. The test outcome can be positive which is 
the recognition result correctly recognized as drowsy and fatigue condition. While the negative results 
indicate as awake. However, the positive can be further described as true positive (TP) and false positive 
(FP). TP is when the recognition result (drowsy and fatigue) are correctly recognized as the expected result 
(drowsy and fatigue). Meanwhile, FP is when the recognition result (awake) are incorrectly recognized as the 
expected result (drowsy and fatigue). 

Next, the negative can be true negative (TN) and false negative (FN). TN is when the recognition 
result (awake) are correctly recognized as the expected result (awake). On the other hand, FN is when the 
recognition result (drowsy and fatigue) are incorrectly recognized as the expected result (awake). Table 2 
shows the confusion matrix outcome from the experiment conducted. The total tested images are composed 
of 50 images for drowsy and fatigue condition and 50 each for awake condition. 


Table 2. Confusion matrix result 
Predicted Classification 


Drowsy and Fatigue Awake 
Actual Condition Drowsy and Fatigue 46 (TP) 4 (FP) 
Awake 6 (FN) 44 (TP) 


Validation involves calculating four objective measures of test performance, namely, sensitivity, 
specificity, positive predictive value (PPV) and negative predictive value (NPV). 

Next, the validation involves calculating five objective measures of test performance, 
namely, accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value 
(NPV) [25-26] are computed. Accuracy measures the overall performance of the testing. Sensitivity measures 
the proportion of the correct predicted positive classes while specificity measures the proportion of the 
negative classes that have correctly classified [19-20]. Equations (3-5) shows the formula to calculate 
accuracy, sensitivity and specificity respectively. 
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Number of TRUE result 


Accuracy = Total number of testing images x 100 (3) 
Fite EP 

Sensitivity = ae (4) 
Ey tery TN 

Specificity = wae (5) 


3. RESULTS AND ANALYSIS 

Table 3 shows the summarization of the results results achieved by the study including the positive 
prediction value (PPV) and negative prediction value (NPV). Based on the experiment conducted, the 
accuracy achieved by this study is 90.00%. On the top of that, sensitivity and specificity rate achieved are 
0.885 and 0.917 respectively. 


Table 3. Summary result 


Measurement Result 
Accuracy 90.00% 
Sensitivity 0.885 
Specificity 0.917 
Positive Prediction Value (PPV) 0.920 
Negative Prediction Value (NPV) 0.880 


4. CONCLUSION 

This study proposed a real-time recognition of drowsy and fatigue facial expression. It demonstrates 
a promising result in recognizing drowsy and fatigue face expression of the driver using the Viola-Jones 
algorithm, shape-based features and support vector machine (SVM) with an accuracy of 90.00%. It is also 
stated that this study achieved a positive prediction value (PPV) as 0.920 and negative prediction value 
(NPV) as 0.880. Besides, both the sensitivity and specificity achieve a promising value of 0.885 and 0.917. 
Both of the sensitivity and specificity values obatianed is high and it can be concluded the proposed study 
able to distinguish between drowsy and fatigue expression and awake face expression accordingly. However, 
it is believed that the hybridization or integration of any existing techniques for both feature extraction and 
classification can improve the accuracy result in future. 
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